Let's review what we
learned so far:
- compiler version need to be identical and recorded
- build options and their order needs to be identical and recorder
- build path needs to be identical and recorded
(otherwise debug symbols - and BuildIDs - change)
- diffoscope helps checking for differences in build output
We stopped when compiling a PE .exe produced a varying output.
It turns out that PE carries a build date timestamp.
The
spec says that bound DLLs timestamps are refered to in the "Delay-Load Directory Table". Maybe that's also the date Windows displays when a system-wide DLL is about to be replaced, too.
Build timestamps looks unused in .exe files though.
Anyway, Stephen Kitt pointed out (thanks!) that Debian's MinGW linker
binutils-mingw-w64 has an upstream-pending patch that sets the timestamp to
SOURCE_DATE_EPOCH if set.
Alternatively, one can pass
-Wl,--no-insert-timestamp to set it to 0 (though see caveats below):
$ i686-w64-mingw32.static-gcc -Wl,--no-insert-timestamp hello.c -o hello.exe
$ md5sum hello.exe
298f98d74e6e913628a8b74514eddcb2 hello.exe
$ /opt/mxe/usr/bin/i686-w64-mingw32.static-gcc -Wl,--no-insert-timestamp hello.c -o hello.exe
$ md5sum hello.exe
298f98d74e6e913628a8b74514eddcb2 hello.exe
If we don't care about debug symbols, unlike with ELF, stripped PE binaries look stable too!
$ cd repro/
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
$ cd ..
$ cp -a repro repro2/
$ cd repro2/
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
Now that we have the main executable covered, what about the dependencies?
Let's see how well
MXE compiles SDL2:
$ cd /opt/mxe/
$ cp -a ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp
$ rm -rf * && git checkout .
$ make sdl2
$ md5sum ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
68909ab13181b1283bd1970a56d41482 ./usr/i686-w64-mingw32.static/lib/libSDL2.a
68909ab13181b1283bd1970a56d41482 /tmp/libSDL2.a
Neat - what about another build directory?
$ cd /usr/srx/mxe
$ make sdl2
$ md5sum usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
c6c368323927e2ae7adab7ee2a7223e9 usr/i686-w64-mingw32.static/lib/libSDL2.a
68909ab13181b1283bd1970a56d41482 /tmp/libSDL2.a
$ ls -l ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
-rw-r--r-- 1 me me 5861536 mars 23 21:04 /tmp/libSDL2.a
-rw-r--r-- 1 me me 5862488 mars 25 19:46 ./usr/i686-w64-mingw32.static/lib/libSDL2.a
Well that was expected.
But what about the
filesystem order?
With such an automated build, could potential variations in the order of files go undetected?
Would the output be different on another filesystem format (ext4 vs. btrfs...)?
It was a good opportunity to test the
disorderfs fuse-based tool.
And while I'm at it, check if
reprotest is easy enough to use (the manpage is scary).
Let's redo our basic tests with it - basic usage is actually very simple:
$ apt-get install reprotest disorderfs faketime
$ reprotest 'make hello' 'hello'
...
will vary: environment
will vary: fileordering
will vary: home
will vary: kernel
will vary: locales
will vary: exec_path
will vary: time
will vary: timezone
will vary: umask
...
--- /tmp/tmpk5uipdle/control_artifact/
+++ /tmp/tmpk5uipdle/experiment_artifact/
--- /tmp/tmpk5uipdle/control_artifact/hello
+++ /tmp/tmpk5uipdle/experiment_artifact/hello
stat
@@ -1,8 +1,8 @@
Size: 8632 Blocks: 24 IO Block: 4096 regular file
Links: 1
-Access: (0755/-rwxr-xr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
+Access: (0775/-rwxrwxr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
Modify: 1970-01-01 00:00:00.000000000 +0000
Birth: -
# => OK except for permissions
$ reprotest 'make hello && chmod 755 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
c8f63b73265e69ab3b9d44dcee0ef1d2815cdf71df3c59635a2770e21cf462ec hello
$ reprotest 'make hello CFLAGS="-g -O2"' 'hello'
# => lots of differences, as expected
Now let's apply to the MXE build.
We keep the same build path, and also avoid using
linux32 (because MXE would then recompile all the host compiler tools for 32-bit):
$ reprotest --dont-vary build_path,kernel 'touch src/sdl2.mk && make sdl2 && cp -a usr/i686-w64-mingw32.static/lib/libSDL2.a .' 'libSDL2.a'
=======================
Reproduction successful
=======================
No differences in libSDL2.a
d9a39785fbeee5a3ac278be489ac7bf3b99b5f1f7f3e27ebf3f8c60fe25086b5 libSDL2.a
That checks!
What about a full MXE environment?
$ reprotest --dont-vary build_path,kernel 'make clean && make sdl2 sdl2_gfx sdl2_image sdl2_mixer sdl2_ttf libzip gettext nsis' 'usr'
# => changes in installation dates
# => timestamps in .exe files (dbus, ...)
# => libicu doesn't look reproducible (derb.exe, genbrk.exe, genccode.exe...)
# => apparently ar timestamp variations in libaclui
Most libraries look reproducible enough.
ar differences may go away at FreeDink link time since I'm aiming at a static build.
Let's try!
First let's see how FreeDink behaves with stable dependencies.
We can compile with
-Wl,--no-insert-timestamp and strip the binaries in a first step.
There are various issues (timestamps, permissions) but first let's check the executables themselves:
$ cd freedink/
$ reprotest --dont-vary build_path 'mkdir cross-woe-32/ && cd cross-woe-32/ && export PATH=/opt/mxe/usr/bin:$PATH && LDFLAGS='-Wl,--no-insert-timestamp' ../configure --host=i686-w64-mingw32.static --enable-static && make -j$(nproc) && make install-strip DESTDIR=$(pwd)/destdir' 'cross-woe-32/destdir/usr/local/bin'
# => executables are identical!
# Same again, just to make sure
$ reprotest --dont-vary build_path 'mkdir cross-woe-32/ && cd cross-woe-32/ && export PATH=/opt/mxe/usr/bin:$PATH && LDFLAGS='-Wl,--no-insert-timestamp' ../configure --host=i686-w64-mingw32.static --enable-static && make -j$(nproc) && make install-strip DESTDIR=$(pwd)/destdir' 'cross-woe-32/destdir/usr/local/bin'
--- /tmp/tmp2yw0sn4_/control_artifact/bin/freedink.exe
+++ /tmp/tmp2yw0sn4_/experiment_artifact/bin/freedink.exe
@@ -2,20 +2,20 @@
00000010: b800 0000 0000 0000 4000 0000 0000 0000 ........@.......
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000030: 0000 0000 0000 0000 0000 0000 8000 0000 ................
00000040: 0e1f ba0e 00b4 09cd 21b8 014c cd21 5468 ........!..L.!Th
00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f is program canno
00000060: 7420 6265 2072 756e 2069 6e20 444f 5320 t be run in DOS
00000070: 6d6f 6465 2e0d 0d0a 2400 0000 0000 0000 mode....$.......
-00000080: 5045 0000 4c01 0a00 e534 0735 0000 0000 PE..L....4.5....
+00000080: 5045 0000 4c01 0a00 0000 0000 0000 0000 PE..L...........
00000090: 0000 0000 e000 0e03 0b01 0219 00f2 3400 ..............4.
000000a0: 0022 4e00 0050 3b00 c014 0000 0010 0000 ."N..P;.........
000000b0: 0010 3500 0000 4000 0010 0000 0002 0000 ..5...@.........
000000c0: 0400 0000 0100 0000 0400 0000 0000 0000 ................
-000000d0: 00e0 8900 0004 0000 7662 4e00 0200 0000 ........vbN.....
+000000d0: 00e0 8900 0004 0000 89f8 4e00 0200 0000 ..........N.....
000000e0: 0000 2000 0010 0000 0000 1000 0010 0000 .. .............
000000f0: 0000 0000 1000 0000 00a0 8700 b552 0000 .............R..
00000100: 0000 8800 d02d 0000 0050 8800 5006 0000 .....-...P..P...
00000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000120: 0060 8800 4477 0100 0000 0000 0000 0000 . ..Dw..........
00000130: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000140: 0440 8800 1800 0000 0000 0000 0000 0000 .@..............
stat
@@ -1,8 +1,8 @@
Size: 5121536 Blocks: 10008 IO Block: 4096 regular file
Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
-Modify: 2017-03-26 01:26:35.233841833 +0000
+Modify: 2017-03-26 01:27:01.829592505 +0000
Birth: -
Gah...
AFAIU there is something random in the linking phase, and sometimes the timestamp is removed, sometimes it's not.
Not very easy to track but I believe I reproduced it with the "hello" example:
# With MXE:
$ reprotest 'i686-w64-mingw32.static-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -Dmain=SDL_main -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
# => different
# => maybe because it imports the build timestamp from -lSDL2main
# With Debian's MinGW (but without SOURCE_DATE_EPOCH):
$ reprotest 'i686-w64-mingw32-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -Dmain=SDL_main -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
0b2d99dc51e2ad68ad040d90405ed953a006c6e58599beb304f0c2164c7b83a2 hello
# Let's remove -Dmain=SDL_main and let our main() have precedence over the one in -lSDL2main:
$ reprotest 'i686-w64-mingw32.static-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
6c05f75eec1904d58be222cc83055d078b4c3be8b7f185c7d3a08b9a83a2ef8d hello
$ LANG=C i686-w64-mingw32.static-ld --version # MXE
GNU ld (GNU Binutils) 2.25.1
Copyright (C) 2014 Free Software Foundation, Inc.
$ LANG=C i686-w64-mingw32-ld --version # Debian
GNU ld (GNU Binutils) 2.27.90.20161231
Copyright (C) 2016 Free Software Foundation, Inc.
It looks like there is a random behavior in binutils 2.25, coupled with SDL2's wrapping of my
main().
So FreeDink is
nearly reproducible, except for this build timestamp issue that pops up in all kind of situations.
In the worse case I can zero it out, or patch MXE's binutils until they upgrade.
More importantly, what if I recompile FreeDink
and the dependencies twice?
$ (cd /opt/mxe/ && make clean && make sdl2 sdl2_gfx sdl2_image sdl2_mixer sdl2_ttf glm libzip gettext nsis)
$ (mkdir cross-woe-32/ && cd cross-woe-32/ \
&& export PATH=/opt/mxe/usr/bin:$PATH \
&& LDFLAGS="-Wl,--no-insert-timestamp" ../configure --host=i686-w64-mingw32.static --enable-static \
&& make V=1 -j$(nproc) \
&& make install-strip DESTDIR=$(pwd)/destdir)
$ mv cross-woe-32/ cross-woe-32-1/
# Same again...
$ mv cross-woe-32/ cross-woe-32-2/
$ diff -ru cross-woe-32-1/destdir/ cross-woe-32-2/destdir/
[nothing]
Yay!
I could not reproduce the build timestamp issue in the stripped binaries, though it was still varying in the unstripped
src/freedinkedit.exe.
I mentioned there was other changes noticed by diffoscope.
- Changes in file timestamps.
That one is interesting.
Could be ignored, but we want to generate an identical binary package/archive too, right?
That's where
archive meta-data matters.
make INSTALL="$(which install) install -p" could help for static files, but not generated ones.
The doc suggests clamping all files to SOURCE_DATE_EPOCH - i.e. all generated files will have their date set at that timestamp:
$ export SOURCE_DATE_EPOCH=$(date +%s) \
&& reprotest --dont-vary build_path \
'make ... && find destdir/ -newermt "@$ SOURCE_DATE_EPOCH " -print0 xargs -0r touch --no-dereference --date="@$ SOURCE_DATE_EPOCH "' 'cross-woe-32/destdir/'
- Changes in directory permissions
Caused by varying umask.
I attempted to mitigate the issue by playing with
make install MKDIR_P="mkdir -p -m 755" (1).
However even
mkdir -p -m ... does not set permissions for intermediate directories.
Maybe it's better to set and record the umask...
So, aside from minor issues such as BuildIDs and build timestamps, the toolchain is pretty stable as of now.
The issue is more about fixing and recording the build environment.
Which is probably the next challenge
Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.
In July, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 25.25h for LTS (out of 30 max; all done) and 13.25h for ELTS (out of 20 max; all done).
We shifted suites: welcome Stretch LTS and Jessie ELTS. The LTS->ELTS switch happened at the start of the month, but the oldstable->LTS switch happened later (after finalizing and flushing proposed-updates to a last point release), causing some confusion but nothing major.
ELTS - Jessie
Also this was the last month before shifting suites: goodbye to Jessie LTS and Wheezy ELTS, welcome Stretch LTS and Jessie ELTS.
ELTS - Wheezy
I know most Debian people know about this already But in case you
don t follow the usual Debian communications channels, this might
interest you!
Given most of the world is still under COVID-19 restrictions, and that
we want to work on Debian, given there is no certainty as to what the
future holds in store for us Our DPL fearless as they always are
had the bold initiative to make this weekend into the first-ever
So, we are already halfway through DebCamp (which means, you can come
and hang out with us in the 
A minor routine update 0.0.4 of 
Here is my monthly update covering what I have been doing in the free software world in October 2017 (
How to change irssi's timezone without restart
Dear conference organizers,
In this day and age, people stream conferences and other events over the
Internet. Most of the Internet happens to be in a different timezone from
yours (it's crazy, I know!). This means that if you publish a schedule,
please say which timezone it's in. We've even got this thing called
JavaScript now, which allows you to also convert times to the user's local
timezone (the future is now!), so you might want to consider using it. :-)
(Yes, this goes for you, 
I've heard about 3d-printing a lot in the past, although the hype seems to have mostly died down. My view has always been "That seems cool", coupled with "Everybody says making the models is very hard", and "the process itself is fiddly & time-consuming".
I've been sporadically working on a project for a few months now which displays tram-departure times, this is part of 
A book about being woman, mother in a modern but still traditional society. About happiness and fulfillment, love and sex, responsibility and dependency, about life: 
Written by a friend back from my old times in Vienna,